-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Add leader election to controller manager #118
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Yeah, it's probably a decent idea. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is configmap-based leader election the reccomended default these days? I'm fine with us choosing an opinionated option, but I want to make sure it's the right one, and we should document that.
Also, we should choose a sane default for the configmap id (and potentially namespace) if we can.
I have refactored the test suite to reduce the duplication. I asked in Slack in #SIG-api-machinery and got confirmation that configmaps are preferrred over endpoints, however in 1.12 there will be a dedicated object coming for this purpose, so perhaps make a TODO for upgrading it once the controller-runtime gets upgraded to 1.12. Where would you default the ID and the namespace? I imagined these would be defaulted by adding some flags to wherever the manager is initialised (one of the templates in controller-tools), mine looks like this at the moment:
Where those variables come from flags. |
Yes, please add the TODO (
Namespace should default to the current namespace, when we can figure out it (e.g. via some default downward API environment variable). ID should have some sane default assuming this is the only controller in the namespace, and then we can make it overridable if need be. We should endeavor to make it so that the defaults work when using the setups that we describe in our docs and in kubebuilder. |
Added.
I've added a
I've defaulted it to |
ffc3047
to
f22e071
Compare
I like the defaulting logic, I think this looks decent. Does the leader election ConfigMap get created with the ID as the name, verbatim? If so, perhaps something like "controller-leader-election-helper" for the ID. 🚲 🏠 |
Yes, precisely. If the ID is |
return nil, err | ||
} | ||
|
||
// TODO(JoelSpeed): switch to leaderelection object in 1.12 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm curious to know more about this but am having a hard time finding other references to it. Do you have any links to info about this object?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I asked about it in #sig-api-machinery, and got the info from there: https://kubernetes.slack.com/archives/C0EG7JC6T/p1535318981000100
I know nothing more about it, sorry
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I got this reference from that slack thread: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/coordination/v1beta1/types.go#L27
7a08cd5
to
666f61c
Compare
There were merge conflicts so I've just rebased on top of master |
666f61c
to
38fc6fb
Compare
589c137
to
7fa750f
Compare
I fixed the tests that were failing, I think this might be ready to go? PTAL @DirectXMan12 @pwittrock |
Yeah, sorry, trying to do one last scan-through before lgtm. |
To be clear, when we lose the leader election lock, we exit out, and presumably the pod gets restarted? Can you add a note in the |
@DirectXMan12 I've added a note about the behaviour and what to expect when a leader loses leadership. Does that seem sufficient? |
Perfect. Can you squash down into "logical" commits? We can merge after that. |
c034f1f
to
50b6bdd
Compare
@DirectXMan12 Squashed as per your request 😄 |
/lgtm |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: DirectXMan12, JoelSpeed The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Add leader election to controller manager
Fixes #93
This is a first pass at implementing leader election code into the manager.
I've created a
leaderelection
package which contains constructors for a resourcelock.Interface as well as a fake for use in testing.I've added three new options to the manager options for leader election, a
bool
to enable the feature (disabled by default to preserve behaviour) and a name and namespace for the configmap that it will use for the lock.I was considering factoring out the
Start
test suite such that you can run it with differentOptions
, since that's all I do in the tests for the leader election and theres a lot of duplication of the test suite. Does that seem like a reasonable thing to do?